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An integrative causal process model of evaluation 
utilization variables is presented. The model was developed through a 
traditional approach to literature review that lists results from 
published studies and relates these to the research topic, and 
through an approach that tries to integrate the models found in the 
literature search. Meta-model ing means developing models from models. 
It is a non-quantitative inductive theory-building approach done on 
published research studies as a specific source of data. Variables 
that affect utilization were identified, and implicit-cause and 
explicit-cause models in the literature were located. Reviewing these 
implicit and explicit models identifies * he following key variables 
as facilitators of utilization: (1) participation; (2) organizational 
process and communication; (3) feedback; (A) politics and 
self-interested decision making; (5) use management; and (6) 
cognitive and behavioral uses. Evaluation usage is a continual 
process that evolves and changes shape over time, which is dependent 
on local contextual, organization, and political dimensions, in which 
participation by stakeholders and feedback seem especially useful. 
The proposed model is presented in Figure 7. Six additional figures 
illustrate the discussion. Contains 74 references. (SLD) 
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Several key themes have emerged from debates in the evaluation 
literature during the past ten or fifteen years that have implicating for 
current conceptions of utilization* The first debate surrounds the issue of 
evaluation as a summacive product versus evaluation as a "theory dri en" 
process and product. It has been suggested that evaluation without "theory" 
is like evaluating a "black box" (Chen & Rossi, 1980, 1987, 1989, 1992); 
therefore, these authors contend, the theory driven approach is needed. This 
approach involves developing "models" of programs so that 1) the internal 
dynamics of a program can be understood, and 2) generalizable evaluation 
theory can be developed. Recent thinking in the utilization literature seems 
to be nosing in the direction of theory-driven evaluation (Chen, 1990; Chen & 
Rossi, 1989; Huberman, 1987; Huberman & Cox, 1990). 

The second debate surrounds the issue of instrumental use versus 
enlightenment; the former means using evaluation results to make concrete 
decisions about a specific program, and the latter means changing one's 
thinking about programs. The most prominent advocates of these positions are, 
respectively, Patton (1986, 1988), who suggests that a great deal of 
instrumental use does occur, and Weiss (1984, 1988), who suggests that most 
evaluation use occurs over the long-term after any single evaluation has long 
past. Current "resolution" of this issue suggests that both kinds of 
evaluation occur and both are important (Alkin, 1990) • 

Another debate surrounds the issue of the "social responsibility" of an 
evaluator. It has been suggested that decisions about programs should come 
from managers and policy-makers, not evaluators (e.g., Wholey, 1983; Patton, 
1986; Shadish, Cook, & Leviton, 1991). In this tradition, the evaluator 
becomes an expert who provides information but does not tell policymakers what 
to do with it. Shadish et al. (1990), calling evaluation theory which 
advocates a value position "prescriptive theory, " also contend that value 
positions "should" be avoided. 

Others have suggested that an evaluator has a social responsibility when 
offering recommendations. For example, Scriven would contend that the 
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evaluator has a moral responsibility to act in the interest of a "better 
society, M in particular, a society with a lower degree of income and wealth 
inequality (Scriven, 1983). Furthermore, advocates of the qualitative 
evaluation paradigm (as contrasted with the posit ivist paradigm) have argued 
for the realization that evaluation always involves some kind of value 
judgement. These kinds of issues about valuing and questions about who makes 
the decisions are important to researchers in the utilization literature. 
Specifically, directing attention to the evaluator 's role and approach has 
implications for who and how results are utilized. 

Another debate is about stakeholders' roles in the evaluation process. 
Patton has long contended that certain people in an organization should work 
closely with the evaluator and act as advocates for the evaluation in the 
organization (Patton, 1978, 1986). These advocates will usually be high level 
decision makers. Wholey has argued for getting managers more involved and, 
essentially, learning to "manage for use* 4 ; furthermore, if an evaluation has 
little chance of being used then Wholey would say "don't do it" (Wholey, 
1983). Alkin, points to the importance of "primary users" (1991). As can be 
seen in these views, currently, "participatory" evaluation approaches are 
viewed with high esteem, and participation is viewed as one of the key 
influences on stakeholder satisfaction and utilization (Cousins & Earl, 1992). 

One might surmise from this review that the utilization literature is 
dynamic, but, at the same time, the following question can be asked: do we 
know more today about utilization than we did ten years ago? While the answer 
seemingly is H yes," this does not mean that the relative importance of 
variables related to utilization is known, or that the degree of utilization 
can be predicted with much accuracy. One problem is that while there are many 
lists of variables known to affect utilization, there are not many models of 
evaluation utilization that integrate variables into systems, showing their 
interrelationships . It appears, therefore, that there is a need for 
additional research which integrates past literature and offers more holistic 
findings and predictions. New research should address this need. 
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One way to meet this need in the utilization literature is by developing 
integrative cause models, 'Tiusal process models should be a welcome addition 
to the literature as researchers attempt to order the variables that affect 
utilization. In other words, rather than just suggesting that single 
variables affect utilization, it would be helpful if researchers looked for 
strings of variables, showing indirect effects and intervening mechanisms. 

Integrative models should also help advance utilization theory by 
beginning to explain how the utilization variables operate together. Models 
should help evaluators communicate what is known about evaluation use in a 
more holistic way, which may increase the use of research findings, in time, 
explanatory models with greater predictive power might be developed. It is 
contended here, therefore, that modeling of utilization variables is the next 
logical for the evaluation utilization literature. it is the purpose of this 
paper, therefore, to offer an integrative causal process model of the 
utilization variables. 

Two approaches to reviewing the literature were used in developing the 
integrative causal process model. First, the traditional approach which lists 
results from various published studies and relates these studies to the 
research topic was used. The second approach, which was especially relevant 
for this study, looks for published theoretical or empirical models and tries 
to integrate these models; this second approach is called, here, meta- 
modeling. The section below called "Variables that Affect Utilization" 
follows the traditional approach; the section below called "Models of 
Evaluation Utilization" follows the meta-modeling approach which is reviewed 
next. 

Meta-Modelinq 

It is contended here that additional and more systematic attempts at 
modeling can be made from other models in the existing literature . Meta- 
modeling, as term is used here, means developing models from models. 
Describing this methodology is seen as important so that, perhaps, the 
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beginning of a more rigorous methodology can be developed . Such a methodology 
for developing models from models is rarely, if ever, discussed in current 
social science methods texts. 

Meta-raodelin$ is a non-quantitative, inductive theory building approach 
done on a specific a data source — published research studies. Specifically, 
implicit cause-models are constructed (i.e., models constructed from ideas 
described in previous research studies), and explicit cause-models are located 
(i.e., already depicted models). Then, systems of concepts, categories, and 
variables are pulled from the implicit and explicit models and integrated into 
a new (holistic) model. In short, the output of meta-modeling is another 
model. 

After an initial meta-model is developed, the process continues 
iterating until the researcher is satisfied that the "best" model had been 
obtained. During this process, new variables are added and others may be 
deleted. The researcher refines the model by returning to the literature and 
examining the models in a new light; he or she may also, at this point, see a 
need to look at models that were not considered relevant during earlier runs. 
Each time the meta-model is changed, it is checked for its "fit" with 
empirical results in the literature and with the developing theory. During 
the process, one can try variables at different levels of abstraction. In the 
present study, the original model had too many variables; hence, a smaller, 
more abstract model was developed which still retained much of the meaning of 
the more complex model — in short, the model was more parsimonious. 

The meta-modeling approach has some similarities with the type of 
traditional theory building called "causal modeling" in that models are 
developed from previous theory (e.g., Asher, 1983; Davis, 1984; Miles 6 
Huberman, 1984). The traditional criteria for causality are followed: 
association or covariation (to establish relationship between variables), 
temporal order (to establish direction of causality), and elimination of 
alternative explanations or "third variables" (to establish that the variables 
are causally related rather than spuriously related). The approach also draws 
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upon the grounded theory approach, recently succinctly rearticulated by 
Strauss and Corbin (1990); the strategy in both grounded theory and meta- 
modeling ia to search for categories, to describe their properties and 
dimensions, and to atterr^t to order the categories; as a result, the met i- 
modeling approach produces models "grounded" in the literature. For a more 
detailed description of the grounded theor/ approach see Strauss and Corbin 
(1990). Met a-mode ling also has some similarities with the technique called 
"meta-ethnography" where ethnographers construct integrative ethnographies 
based on other published ethnographies (Noblit & Hare, 1988). 

Variables that Affect Utilization 

In an another paper by the present author (Johnson, 1992a) over 100 
variables affecting utilization were identified. Rather than list all of 
these variables here, the reader is directed to the two published literature 
reviews that are currently available (Leviton & Hughes, 1981; Cousins & 
Leithwood, 1986). The variables listed here are categorized by typologies . 

Boyer (1989) categorizes the utilization variables into seven 
categories. These categories are: perceived relevance, timeliness, extent and 
quality of communication, credibility, presentation of findings, advocacy by a 
key individual, and political considerations (Boyer, 1939; Boyer & Langbein, 
1991). Boyer, and Langbein, developed these categories especially for 
decision making in congress and the federal bureaucracy. Information 
obviously has to reach members of congress and their aids before a particular 
law is voted on (instrumental use). Credibility is also important so that 
decisions, often politically motivated, can be justified and defended (i.e., a 
sort of symbolic use) (Chelimsky, 1987). 

Cousins and Leithwood (1986) developed a model of sorts which is 
reviewed in the next section. Here, their frequently cited categorization of 
utilization variables is shown for comparison purposes. They categorize the 
utilization variables into two sets. Set one is called "evaluation 
implementation" and includes: evaluation quality, credibility, relevance, 
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communication quality, findings, and timeliness. Set two, called "decision or 
policy setting- variables, includes: information needs, decision 
characteristics, political climate, competing information, personal 
characteristics, and commitment and/or recept Iveness to evaluation. One 
advantage of the Cousins and Leithwood article, compared to Boyer (1989), was 
the use of the two factor categorization scheme. 

The Leviton and Hughes (1981) typology is as follows: 1) relevance of 
evaluation to needs of potential users, 2) extent of communication between 
potential users and producers of evaluations, 3) translation of evaluations 
into their implications for policy and programs, 4) credibility or trust in 
evaluations, and 5) commitment or advocacy by individual users. 

It is readily seen that there is a good deal of overlap in the three 
typologies just given. Four weaknesses of the typologies are noted: 1) 
organizational characteristics were given little weight, 2) stakeholder 
participation was not emphasized, 3) no ordering of the categories was 
attempted, and •?) "organizational learning" was ignored. In the present 
research, ordering of variables is a major goal. Furthermore, participation 
and organizational learning are key elements in the theoretical framework 
used. 

Models of Evaluation Utilization 

Next some models are drawn from the evaluation utilization literature. 
First, implicit cause-models were identified and, from these, explicit models 
were constructed. A key literature source for these implicit models was the 
excellent evaluation theory text by Shadish et al. (1991); these authors have 
detailed chapters on the thought of Campbell, Scriven, Weiss, Wholey, Stake, 
Cronbach, and Rossi. Models were constructed for these seven theorists. 
After discussing the implicit models, explicit cause-models found in the 
utilization literature are reviewed. 

The first theorist who has an implicit cause-model is Donald Campbell. 
Campbell believes that the major responsibility for use of evaluations lies in 
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the political process not the evaluator. Over time, his -evolutionary 

epistemology" predicts, better programs will survive and tociety will be 

improved (Campbell, 1969, 1984). The evaluator is like a "scientist" 

conducting evaluations with the best methods possible (especially 

experiments); she/he does not, however, directly promote the use of findings. 

Overall, Campbell offers little for a modern theory of utilization ? instead, 

he assumes that it will "just happen." Nonetheless, a depiction of Campbell's 

ideas might look like the following: 

Program evaluation — > Consideration by «> Instrumental 
report policy-makers Use 

and along with 

reports of other 
past programs information 

Michael Scriven, like Campbell, does not offer an explicit theory of 
use. Scriven (1983) advocates a "consumer reports" type model of evaluation, 
where the evaluator examines the comparative strengths and weaknesses of a 
program and makes a final summative judgement of worth — is the program "good" 
or is the program "bad?" He seems to assume that programs are like product*, 
and that consumers will examine strengths and weaknesses and make rational 
choices. However, his model may not apply very well to program evaluation (as 
contrasted with product evaluation). For example, the public sector promotes 
interest group participation in judging actual programs; it's a political 
process involving multiple interests, and which program evaluation is utilized 
at any given time is not always predictable. In other words, multiple 
stakeholders may simply not "buy" the either/or logic of consumer product 
evaluations. To Scriven 's credit, he does advocate comparative evaluations in 
decision making. (One might also question his advocacy of summative rather 
than formative evaluation.) 

A depiction of Scriven 's implicit-cause-model might look like this: 

Program > Final Summative > 

Implementation Evaluation Report 

Marketplace of > Usage by People 

Ideas and Interested in 

Information the Program/Product 
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Carol Weiss (1984), perhaps more than any other prominent theorist 
believes that little or no instrumental use ever occurs roost use, she 
contends, is through enlightenment, she has recently backed down slightly 
about the lack of occurence of instrumental use (see, Alkin, 1990). o.er 
time decision accretion takes place; that is: 

Policies are not made at a single point in time; they seen to happen as 
the result of gradual accretions, the build-up of small choices, the 
closing of small options and the gradual narrowing of available 
alternatives (cited in Shadish 1991: p. 192; also see Weiss, 1980a). 

This kind of use is very difficult to measure because it is conceptual; it is 
not tangible. 

Weiss (1983) does offer a very interesting implicit cause-model of 
aepjBjgri making at the individual level of analysis called "I-I-l Analysis, - 
She says decisions are the result of three major influences: 1) information, 
2) ideology, and 3) interests. The influence of these three factors is 
tempered by the organizational environment in which the person resides (Alkin, 
1990). Furthermore, when making a decision, deci.xon makers conduct "truth 
tests" (i.e., it conforms to prior knowledge) and "utility tests" (i.e., is 
the evaluation feasible and action oriented?) (Weiss, 1980b). 

A more explicit depiction of the process described by Weiss might look 

like: 

Interests 

Truth test 

Organizational > ideology > > Dec ision 

environment Utility test 

Information 

Joseph Wholey, unlike Weiss, developed a more detailed theory of 
instrumental use; in this theory it is suggested that evaluation should 
directly serve the needs of management (Whole/, 1983, 1985). If the potential 
£ ° r uae of an evaluation does not exist (which he would determine from an 
"evaluability assessment") then the evaluation should not be done. Wholey 
(1985) offers advice on how to manage programs and how to ^se evaluative 
information as part of management. In other words, his focus is on 
instrumental use (i.e., immediate tangible use) through effective management. 
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Ho does not question whether a program meets social needs or not — he allows 
managers and policy-makers to make that decision. Wholey's focus is on 
program improvement and effective management, and it is the evaluator's job to 
work with management to improve programs. His approach to management quite 
compatible with modern organization development theory (Harvey & Brown, 1992; 
Carnall, 1990) . 

Wholey rarely recommend* use of the rigorous experimental methods 
advocated by Campbell. Programs are characterized by limited resources and 
purposes, wholey (1979), therefore, recommends a "sequential purchase of 
information." This involves, in order of increasing expense, the following: 

1. Evaluabixity assessment 

2. Rapid feedback assessment 

3. Performance monitoring 

4. Intensive evaluation 

Each of these could probably be modeled. In general, however, a model of 

Wholey's ideas about evaluation use might look like the following: 

Evaluability — > evaluation change in > Continuous 

assessment implementation managers' instrumental 

performance use 

Robert Stake is epistemologically and ontologically a qualitative 
researcher, e.g., he is a relativist. His specific methodology is the case 
study (Stake, 1981). His approach to evaluation, called "responsive 
evaluation," basically means orienting evaluations to program activities and 
stakeholders' needs (Stake, 1975). Through a kind of participatory approach 
to evaluation, a qualitative report using detailed description is produced. 
The evaluator telle a story, and after reading this story (i.e., the report), 
readers are supposed to able to make naturalistic generalizations (Stake, 
1990). That is, readers are said to vicariously experience a program and then 
generalize, based on their experience, to other environments, people, and 
programs. It is noted that this is a kind of conceptual use. The explicit 
cause-model developed from Stake's implicit cause-model of usage is depicted 
as follows: 
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Qualitative — •> Vicarious — > Understanding — > Naturalistic — > 
evaluation experience generalizations 

(case study) 

-~> Conceptual 
Use 

As shown in the model, Stake believes that use of a case study methodology 
ultimately increases evaluation utilization. 

In contrast to Donald Campbell, Lee Cronbach believes that complex 
interactions should be used to describe the nature of reality and that it is 
the evaluator 's job to show in detail the processes going on in a program; the 
main reason for doing this is generalization. For example, Cronbach talked 
about aptitude-treatment-interactions and multiple-variable interactions 
(Cronbach, 1957). He contends that 3, 4, 5 or greater interactions are 
necessary for describing the social world (Cronbach & Snow, 1977). Cronbach 
wants to look at processes going on in a program, and he wants to be able to 
generalize to other, sometimes dissimilar programs, paople, and places, 
(called *UTOS) (Cronbach et al., 1980, 1982). 

Cronbach contends that a major purpose of evaluation is for conceptual 
use (or enlightenment). The evaluator may wish to revaal findings no 
stakeholders continuously during an evaluation (Cronbach et al., 1980) The 
evaluator should "hang around" an evaluation site. In general, results should 
decrease stakeholder uncertainty about the operation of the program; the 
tvaluator is to carry out an "education" role. Because much use of evaluation 
occurs in the long term for programs other than the one being evaluated , 
programs will benefit most from accumulated theoretical knowledge. Finally, 
Cronbach has advocated the use of "standing committees of experts" to 
synthesize findings from evaluation research. 

A tentative depiction of Cronbach' s implicit-cause-model of utilization 
is as follows: 

Analysis of -> Program — > Continuous -> Formative -> Long term 
background Develop- Feedback use Conceptual 

theoretical ment and use 

literature modification 

of program 

and evaluation 

questions 
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Rossi, according to shadish at al. (1991), essentially has no theory of 
use. They say: 

The Leviton and Hughes (1981) conceptualization of use seems tacked on 
to Rossi's theory of practice as an afterthought ... .Rossi may not mean 
to discount the value of instrumental use, but his lack of clarity about 
when it should be pursued leaves him vulnerable to such criticism 
(pp. 423-424) . 

They further suggest that Rossi's (and Chen's, 1983) theory-driven-approach to 
evaluation "subordinates user information needs to social science theory 
needs" (Shadish et al., 1991:p.423). 

It is contended here that Rossi does have an implicit theory of use. 
Perhaps it ia not well developed, but part of his program model does refer to 
use. To increase use, Rossi suggests that evaluators "tailor" evaluation 
activities to local needs (Rossi & Freeman, 1989). In general, how this is 
done depends on the stage and kind of program that is being evaluated. This 
process of "fitting evaluations to programs" can be viewed as an approach co 
increasing evaluation usage. Apparently if one were to conduct the wrong type 
of evaluation it would be "useless." Further, it is predicted that Rossi 
would suggest that forming a congruence between evaluation and need testers 
usage. Finally, Rossi's calls for using theory can also be seen as 
facilitating evaluation use because it allows one to know what is going on in 
the "black box" and, therefore, to do something about the operation of the 
program. 

A tentative depiction of Rossi's ideas is as follows: 

Do your — > Work with — > Collect — > Compare — > modify 

homework, program Data model program 

i.e., review managers with based 

literature and develop reality on 

on similar program model findings 

programs ( instru- 

mental use) 

One last implicit cause-model from the evaluation utilization literature 
was developed from a 1984 paper by Carol Weiss. Weiss presented her ideas in 
the matrices shown in Figure 1 on the next page. 
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FIGURE 1 





Intellectual/ 
Cognitive Domain 


Social /Structure 
Domain 


From Policy 
to 

Evaluation 
Formulation 


A 

Specification of the 
evaluation problem 


B 

Institutional pressures 
on policy makers and 
evaluators that affect 
specification 


From 

Evaluation 
Findings to 
Action 


C 

Interpretation of 
evaluation findings 


D 

Application of 
evaluation to policy 






(From Weiss, 1984) 




Intellectual/ 
Cognitive Domain 


Social /Structure 
Domain 


From Policy 
to 

Evaluation 
Formulation 


A 

Plan with users in 
mind 

Stay close to the scene 


B 

Plan with users 
Concentrate on 
manipulable 
variables 


From 

Evaluation 
Findings to 
Action 


C 

Clear well-written report 
Dissemination: get the 
results out 
Evaluation synthesis 
High quality research 


D 

Translate findings into 
recommendations 
Make recommendations 
feasible to implement 



(From Weiss, 1984) 



After dividing the evaluation process into roughly three stages (including: 1) 
evaluation formulation, 2) conduct of the study, and 3) implications for 
decision), Weiss suggests that evaluators can make the largest impact during 
the first and third stages. She contends that there are two major obstacles 
to use: 1) intellectual and 2) social and structural. In the first matrix, 
Weiss shows the problems or issues that occur for the first and third stages 
crossed with the two obstacles. In the second matrix, she lists what an 
evaluator can do and when he or she should do it. This process might be 
modeled as follows: 



I* Intellectual/Cognitive Domain 

Stakeholder 
participation 

in planning Evaiuator Evaluator 

and specification conducts the writes clear 

of problem > study — «— > report — > ;se 

(Evaluation (Conduct of (Implications 

f orwulation) the study) for decisions) 

II. Social/Structural Domain 

Evaluator 

Concentrate on Evaluator translates 

manipulable conducts the findings into 

variables > study > recommendations — > Use 

(Evaluation (Conduct of (Implications 

formulation) the study) for decisions) 

None of the models reviewed so far has been empirically tested using 
causal modeling techniques (to my knowledge). In fact, the authors have not 
used schematics to depict their models; their models are what have been called 
i mplicit cause-models in the present study. In an implicit cause-model, a 
process is implied, but, generally, is not directly discussed or depicted. 
Some more explicit and empirical models that have appeared in the literature 
now follow. 

A commonly cited model of evaluation utilization was developed by 
Jennifer Green (1988; shown in Figure 2). She also empirically tested the 
model using a qualitative methodology (i.e., two field studies). She studied 
two local program evaluations — a youth employment program and a day-care 
information and referral program. She suggested that stakeholder 
participation is an effective way to promote evaluation use. Based on the 
program data, Green categorized stakeholders into three groups: 1) VIPs (very 
involved person), 2) SIPs (somewhat 

or sometimes involved person), and 3) MIPs (marginally involved person). 
According to the participatory approach, a major goal for the evaluation is to 
get people as involved as possible as soon a possible; hence, VIPs are to be 
desired (Green, 1988) • 

As can be seen in Figure 2, Green views participatory evaluation as a 
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three stage process: 1) the "participatory evaluation process," 2) "uses of 
the evaluation process," and 3) the "process of utilization." Elements in 
stage one are said to affect elements in stages two and three; elements in 
stage two are said to affect th< "greater understanding" part of the stage 
three utilization process. One interesting feature of this model is that it 
includes indirect effects . Also, it operationalizes utilization as a process. 
It would be interesting, possibly, to further test this model (i.e., verify 
it) using quantitative and qualitative modeling techniques. 



FIGURE 2 



Participatory Evaluation Process 



Elements Designed 



Iterative, ongoing 
communication and 
dialogue (especially via 
personal contacts) 



Stakeholders' substantive 
decision making role and 
responsibilities 



Dimensions Experienced 



Cognitive 

Ongoining, active discussion 
ind processing of informa- 
tion related to key program 
issues, amidst diverse 
perspectives and toward 
some decision or action 



Uses of the 
Evaluation Process 



Learning more about the 
program and agency 



Learning more about 
evaluation 



Affective 

Individual feelings of worth 
and value 





Political 

Voice to the less powerful 
Interest and attention from 
the more powerful 


Diversity of stakeholder 
participants 






Contributions of the Evaluation 
Process to Utilization 




Greater understanding^ 
the results 



Heightened perceptions 
of the results as valid, 
credible, persuasive 



Greater acceptance/ 
ownership of the results 



Greater sense of responii. 
bUlty and obligation to 
follow through on the results 



Another utilization model from the evaluation utilization literature was 
done by Cousins and Leithwood (1986; see Figure 3). Because the article in 
which the model appears is a literature summarization, the data for the model 
are from previous research studies. Basically, the model divides twelve 
utilization factors into two more abstract factors called "evaluation 
implementation" and "decision or policy setting." 

In the model, these two abstract factors are shown to influence the 
utilization process (which involves three variables). In other words, this 



14 



lb 



model can be viewed as including two latent independent variables (each 
measured by six variables) which directly influence the evaluation utilization 
process. This model has not been empirically verified on primary data. 

FIGURE 3 




Evaluation utilization framework. 



Another set of models is given by Jean Wollenberg (1986; see Figure 4). 
Wollenberg studied the use of evaluation in two school districts using field 
work methodology over an extended period (i.e., a complete school year). In 
this study, extensive qualitative data were collected. The data were analyzed 
in three ways: 1) using Alkin's 1979 categories, 2) through "forms of use" 
(including direct use, legitimative use, persuasive use, conceptual use, and 
anarchic use), and 3) through "time periods/cycles of program 

implementation/growth" (i.e., data were categorized into the three cycles, in 
effect creating a time-process variable) . The three cycles of program growth 
include a conceptual stage, a developmental stage, and an institutional stage. 

Another interesting model developed by Wollenberg (1986; see Figure 4), 
is useful because it depicts the influence of two different kinds of 
organizational structures on utilization. The two structures are, first, a 
"loosely-coupled" system (i.e., a decentralized, bottom up system) and, two, a 
more tightly controlled, but smaller, system (i.e., a more traditional 
bureaucratic hierarchial form). In the loosely coupled system the evaluator 
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communicated mainly with a bilingual director and teachers (see diagram for 
Montemar School District); the bilingual director played a " gatekeeping" role 
(i.e., deciding who obtains information about the evaluation). In the 
H tighter H district {i.e., Alvarado Union School District), the evaluator 
was the gatekeeper, and, hence, communicated with all of the major stakeholder 
groups. The main use of these two models is, I believe, in showing who 
communicates with whom in the overall organization ; this structural variable 
{i.e., coupling) affects communication, which affects utilization (note the 
implied cause model). 

Figure 4 
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Boyer and Langbein (1991) carried out a quantitative study using 
multiple regression analyses to compare the importance of variables known, 
from previous literature, to influence utilization. These researchers 
examined utilization of evaluation reports by members of congress and key 
congressional staff-persons. Recommendations was the unit of analysis. The 
authors suggested that factors operating in a legislative environment are 
unique to that environment, i.e., not operating in other environments such as 
educational, business, or, in other branches of the federal government (i.e., 
executive and judicial); hence, the purpose of the study. The study was based 
on a sample size of 100 congressional staff who had direct evaluation-use 
experience. 

The results were based on regression equations. Remember, when using 

multiple regressions for exp lanatory purposes, one is in effect using a model 

of direct effects, i.e., it is hypothesized that each of the explanatory 

variables has a direct effect on the outcome variable (controlling for the 

other explanatory variables). Before conducting the regression analyses, 

Boyer and Langbein found that all of the independent variables had significant 

bivariato relationships with utilization. As seen in the regression tables in 

the Boyer et al. article, some of the explanatory variables which were 

important were: 

— Character of the timing, 
— Clarity, 

— Absence of a detractor, 

--Communication, 

— Clarity, 

--Relevance to Congress, 

— Credibility of methodology, and 

— Reputation of performer. 

Another quantitative study of evaluation utilization was done by William 
Johnston for his dissertation (Johnston, 1986; also Johnston, 1988). In this 
study, Johnston drew arrows depicting his assumption of direct effects. One 
model was as follows: 

Benefits Organization > 

Acceptance 

Methodology factor < > 

Variable 

Influence factor > 
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Johnston reported nine empirical models in his dissertation; for each model, 
ha changed one or more of the outcome or explanatory variables. The above 
model is given only as an example. 

Johnston found that "type of change" was the bust predictor of 
evaluation utilization. In order from easiest to achieve utilization to 
hardest, the change dimensions were: 

— behavior changes 

— changes in rules 

— organizational structure changes 

— changes in goals and purposes. 

Thfe other two variables found to be significant pr^^.ctors of utilization were 
influence (i.e., pressure from politically powerful external agencies) and 
study methodology (e.g., conceptualization, generalizability, reliability and 
validity, use of literature, etc.). 

In the next study reviewed, Michael Huberman developed a series of 
models to depict the process of research utilization (Huberman, 1987, 1990). 
He developed a "general model" which integrates his other models, which were: 
1) an "organizational model: researchers" (showing how researchers influence 
the dissemination and utilization process), 2) an "organizational model: 
users" (showing how users of research are influenced by dissemination effort 
and the "predictors of local impact/use), and 3) a "dissemination effort model 
(which showed the sets of variables affecting dissemination efforts). 
Huberman 's general model is shown in Figure 5 (from Huberman, 1987, 1990). 

An can be seen in the figure, Huberman' s explicit cause-model is by far 
the most extensive of the models reviewed here. The model was used to depict 
the causal process operating in a series of local projects that were part of a 
"national program" created by the Swiss National Research Council. In the 
1987 article, the projects were ongoing and the "integrative" models were used 
as preliminary depictions of project processes. In the 1990 article, the 
projects had been completed, and Huberman used a "multiple-case, tracer study" 
design. Basically, Huberman followed the eleven projects (from a population 
of 25 projects) from beginning to 18 months after completion. 

Huberman' s results showed, empirically, that linkages between 
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researchers and practitioners were important for utilization (especially 
conceptual). Furthermore, he found that new collaborative relationships 
frequently developed as a result of practitioner participation in the 
evaluation. It was important that practitioners participates i n the 
evaluation before its conduct, during its conduct, and at the end. 
Participation during the ongoing evaluation was especially important. 
FIGURE 5 
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The last explicit cause-model reviewed here was developed and 
empirically tested in 1980 by Knowlton Johnson. In many ways, Johnson's 
research is similar to the present study. That is, he developed a model from 
the empirical and theoretical literature and he attempted to empirically 
verify the model. This la the only path analytic gtudv of evaluation 
utilization that was found in the utilization literature . To empirically 
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teat the model, Johnson, collected data from 75 decision makers who worked for 
25 organizations. The decision makers has been exposed to "1 or 2 of 19 
evaluation products producad by university personnel and students" (Johnson, 
1980). In-person interviews ffrid paper-and-pencil questionnaires were used for 
data collection. 

Johnson's model is shown in Figure 6 with the beta weights on each 
arrow. Each beta weight shows the standard deviation change in the variable 
receiving the arrow (i.e., an endogenous variable) given a one standard 
deviation change in the causal/ independent variable. 

As seen below, contact and involvement was the most important influence 
on evaluation use (using weights as the "relative importance" indices). 
Transfer intensity and compatibility of results also had direct influences on 
evaluation. Linkage roles had a clear i ndirect influence on evaluation use 
through direct effects on two intervening variables of evaluation use (i.e., 
transfer intensity and compatibility of results). 
FIGURE 6 
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The Prop osed Theoretical Model 

In reviewing the implicit and explicit models of evaluation utilization, 
several kev categories seemed to emerge as facilitators of utilization. 
Participation by program participants/practitioners consistently seemed to be 
important. Some properties of participation seem to include type of 
participation (e.g., autocratic or democratic) and degree of participation 
(varying from great to small). Organizational process and communication also 
seemed to be influential. Possible properties or subcategories here are 
quality of communication (clear/not clear), timeliness of communication, and 
dissemination (e.g., was feedback given during the program or at the end as in 
a traditional report), and type and direction of communication (vertical, 
horizontal, and diagonal). Feedback , which is probably part of the 
communication variable just listed, also appeared to be important (e.g., 
dimensions could be timeliness, frequency, and consensus building). Politics 
and self-interested decision making were important utilization factors (e.g., 
including self-interest, ideology, utility, and power). Use management was 
another important variable (e.g., evaluability assessment, appropriate 
methodology, management commitment). 

Finally, a multiple conceptualization of the outcome variable 
(evaluation utilization) seemed useful, with cognitive use and behavioral use 
being especially important. This conceptualization of cognitive and 
behavioral use seems more in line with social psychological theory. 
Specifically, cognitive awareness of an evaluation will precede behavior 
resulting from the evaluation (unless one wishes to consider unconscious 
processes). Cognitive use is viewed as including awareness of the evaluation 
and the development of attitudes, beliefs, and opinions about a program as a 
result of an evaluation; these cognitive schema may affect behavior toward 
particular programs being evaluated or behavior toward future programs via 
enlightenmnt. Behavioral use involves action, and is closely aligned with 
instrumental use, but may also include symbolic use and legitimative use 
(Owen, 1992). 
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It seams clear that evaluation usage is a continual process that evolves 
and changes shape over time. Models showing relationships among variables, 
therefore, are needed, and should, if possible, allow for change and temporal 
development. The general theme from the meta-modeling analysis done here goes 
like this: evaluation utilization is a continual and diffuse process that is 
dependent on local contextual, organizational, and political dimensions, but 
participation by program stakeholders and continual (two-way) feedback of 
results between evaluator and user seems especially helpful in increasing use 
by increasing evaluation relevance, modification of the evaluation, and 
stakeholder ownership of results. Strauss and Corbin (1990) call a general 
theme, like this, a "story line. 0 

The model resulting meta-model is shown in Figure 7 (on the following 
page). Several concluding observations can be made about this explicit cause- 
model. Participation is hypothesized to be influenced by three variables: 
organizational characteristics, stakeholder/individual characteristics, and 
evaluator characteristics. In particular, participation is expected to be 
highest for organic organizational forms (as contrasted with mechanistic), for 
change-oriented individuals (as contrasted with "bureaucrat ic" individuals), 
and for person-focused evaluators (as contrasted with research-focused). 
Dissemination is affected by participation and by the three variables 
affecting participation. In an organic organization, informal networks are 
common and resulting is a good deal of "grapevine" dissemination. A research- 
focused evaluator will likely view a final report as sufficient, while a 
person-focused evaluator will likely view informal communication before, 
during and after an evaluation as useful. Finally, change-oriented 
individuals will become more interested *n the evaluation (because it may 
represent positive change) and therefore, informally and formally disseminate 
information and results. 
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FIGURE 7 
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Definitions of variables in Figure 7. 
Behavioral use. 

Once becoming aware of an evaluation, behavior or action may result. This behavior may include instrumental, 
symbolic, and/or legit imative uses. 

Cognitive use. 

Degree to which people involved in or directly related to the program are aware of the evaluation, think about the 
information, and form attitudes, beliefs, and opinions about the program being evaluated. 

Competing information. 

Information load in organization that may affect decisions about the program being evaluated: 1) low amount of 
directly competing information, and 2) high amount of directly competing information. 

Dissemination. 

Involves communication of information. This may take place purposely, e.g., via reports and meetings, or it may 
take place informally via social networks. 

Evaluator characteristics. 

The two types ore person focused and research-focused. The person focused evaluator identifies a change agent at 
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the beginning and works with this per eon throughout, making sure that the evaluation answers questions users 
wont answered • 

Individual characteristics. 

Two types of individuals are identified: 1) change seekers, who like to be involved in new programs and enjoy 
learning and change, and 2) ritualists, who have a "bureaucratic personality, 1 * i.e., they like following rules and 
procedures and don't seek much change ♦ 

Interests and ideology. 

Is it in the potential users' self-interest to use evaluation results and recommendations? Measured by it N is in 
his/her interest" or "is not in his/her interest • " 

Organizational characteristics <> 

Organic versus mechanistic types. An organic organization is a relatively flat organization with high levels of 
vertical and horizontal communication. Pov> . r is more in ideas and performance than in position. A mechanistic 
organization is the traditional Weberian bureaucracy where power is located in the position and communication 
travels chains of command shown in an organization chart ( see To si , 1992 ) • 

Participation . 

To what degree (on a ten point scale) people involved in or directly related to the program participate in design 
and conduct of the evaluation, and in dissemination of results. 

Truth and utility tests. 

Do potential users believe evaluation results and that the evaluator is credible, and is the information seen as 

useful in the person's job. Measured by viewed as "true or usefui" versus viewed as "neither true nor useful* " 

organizational learning. 

Involves adjustment on the part of the program or organization, including, for example, changes in the culture, 
strategy, structure, stories, myths, norms, etc. 



Cognitive ise is shown to be directly influenced by "truth and utility 
testing, " consideration of personal "interests and ideology," and "competing 
information" flowing through the organizational environment (Weiss, 1980; 
Cousins & Leithwood, 1986). The three variables influencing participation are 
hypothesized to affect cognitive use indirectly through participation and 
dissemination processes operating within an organization and by a competing 
informational environment. The type of organization, for example, affects the 
nature of participation (who does what, who talks to whom, and who can make 
decisions) resulting in different degrees and types of utilization. In a 
tightly coupled mechanistic organization many potential stakeholders may not 
be aware of the conduct of an evaluation — they never participated, to any 
degree, and they never reached the first step of the utilization process, 
i.e., awareness. Given a mechanistic structure, it may not be in non- 
participators' interests to participate because c 7 their lack of power and 
continued intense job specialization. 
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Finally, organizational learning is shown in the model to result f f :om 
cognitive use and behavioral use. It is well know that organizations change 
and learn over time (e.g., Fiol & Lyles, 1985; Frey, 1990), and utilization 
(e.g., cognitive and behavioral) should affact this process (Cousins & Earl, 
1992). In effect, social constructions of reality in organizations and 
programs may change as a result of a program evaluation (cf* Pitre & Sims, 
1987; Bandura, 1986; Gergen, 1985; Berger & Luckmann, 1967). As participation 
increases, it is hypothesized that use and organizational learning will occur. 
People often enjoy being part of an organizational/program change process and 
will, therefore, pay attention during the evaluation process (Pasmore & 
Fagans, 1992; Johnson, 1992b, Neumann & Hare, 1988). 

It is suggested that future research continue the modeling process as 
done here and elsewhere (e.g., Johnson, 1980; Huberman, 1987). This may, 
perhaps, result in more holistic thinking and more explanatory utilization 
studies. Tosi (1992) offers a nice exemplar for recent model building, 
modeling the operation of person, group, organizational, and environmental 
factors. 

It is also suggested that future research explore potential 
contributions from social science theory (Chen & Rossi, 1987). As an 
example, we might explore what drives the behavior of individual actors and 
stakeholders. The importance of political concerns are already well 
established (e.g., Chelimsky, 1987; Weiss, 1984). We might, however, consider 
using additional theory such as Homans' (1974) and Blau's (1964) "exchange 
theory 11 which suggests that within organizations people "make deals'* and 
social exchanges. Social exchange is probably one of the bases of the 
political processes so frequently identified in the literature. Add to this 
the economic concept of "rational self interest" %nd we further see what 
motivates actors. The resistance to change literature may add insight into 
tendencies operating for order and for change within individuals (e.g., 
Neumann, 1989; Johnson, 1992b; Klein, 1984; Coch & French, 1948). Moving to a 
slightly higher level of abstraction we see that people in organizations are 

25 



ERLC 



27 



constantly performing expected behavioral patterns associated with their rolea 
or positions; on the other hand, behavior is also fluid and emergent as 
expectations develop within specific situations. Finally, the literatures on 
constructivism and constructionism may help explain how different 
constructions of reality develop and operate in different organizations and 
programs (cf. Gergen, 1985; Guba & Lincoln, 1989; Attewell, 1990; Shrura, 1988; 
Bohan, 1990; von Glasserfeld, 1984, 1988). 
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